The Impact of Algorithmic Risk Assessments on Human Predictions and its Analysis via Crowdsourcing Studies

As algorithmic risk assessment instruments (RAIs) are increasingly adopted to assist decision makers, their predictive performance and potential promote inequity have come under scrutiny. However, while most studies examine these tools in isolation, researchers recognize that assessing impact requires understanding the behavior of human interactants. In this paper, building off several recent crowdsourcing works focused on criminal justice, we conduct a vignette study which laypersons tasked with predicting future re-arrests. Our key findings as follows: (1) Participants often predict an offender will be rearrested even when they deem likelihood re-arrest well below 50%; (2) do not anchor RAI's predictions; (3) The time spent survey varies widely across participants cases assessed less than 10 seconds; (4) Judicial decisions, unlike participants' predictions, depend part factors orthogonal re-arrest. These results highlight influence crucial but overlooked design decisions concerns around generalizability constructing analyze impacts RAI